Skip to content

Transform diarized audio into segment-level embeddings

Source: R/2_2_talkEmbedSegments.R
talkEmbedSegments.Rd

Unlike talkEmbed, which returns one embedding per whole audio file, talkEmbedSegments produces one embedding per diarized segment. It takes an audio file together with its diarized transcript (e.g. the output of talkTranscribeDiarise) and embeds each segment, optionally restricting to participant segments.

Usage

talkEmbedSegments(
  audio,
  transcript,
  model = "whisper",
  embeddings = "encoder",
  participant_only = FALSE,
  whisper_model_id = NULL,
  whispa_model_id = "Jarhatz/WhiSPA-V1-Small",
  whispa_repo_path = NULL,
  output_dir = NULL,
  device = NULL,
  condaenv = getOption("talkrpp_condaenv", "talkrpp_condaenv"),
  verbose = FALSE
)

Arguments

audio

(string or character vector) Path to a single audio file (e.g. .wav) or a vector of file paths. Several files are processed separately (paired with `transcript`), returning a named list.

transcript

(string, data.frame, or list) Diarized transcript: a path to a CSV, the transcript tibble returned by talkTranscribeDiarise, or – for several audio files – the named list returned by a multi-file talkTranscribeDiarise() run (or a vector of CSV paths) of the same length as audio. Must contain per-segment start/end timestamps, and a speaker_role column if participant_only=TRUE.

model

(string) "whisper" or "whispa".

embeddings

(string) For model="whisper": "encoder", "decoder", or "both". Ignored for model="whispa".

participant_only

(logical) If TRUE, only segments with speaker_role == "participant" are embedded. Defaults to FALSE because the transcript returned by talkTranscribeDiarise has no speaker_role column; set TRUE only with a transcript that provides one.

whisper_model_id

(string) Optional override of the Whisper model id.

whispa_model_id

(string) WhiSPA checkpoint (model="whispa" only).

whispa_repo_path

(string) Optional path to a local WhiSPA clone, used only if WhiSPA is not pip-installed.

output_dir

(string) Optional directory to also write embedding CSV(s).

device

(string) "cpu", "cuda", or "mps". If NULL, chooses "cuda" when available, then "mps" on real Apple hardware, else "cpu". On virtualized macOS (VMs/CI runners) MPS can silently produce invalid embeddings, so it is never auto-selected there (and explicitly requesting it triggers a warning).

condaenv

(string) Name of the conda environment with the talk stack installed. Defaults to the environment saved by talkrpp_initialize(save_profile = TRUE) (the "talkrpp_condaenv" option), falling back to "talkrpp_condaenv" – so a shared environment (e.g. "text_talk") only needs to be set once via talkrpp_initialize().

verbose

(logical) If FALSE (default), the technical output from the Python backend (model-loading logs, progress bars, warnings) is hidden and only short status messages are shown. Set TRUE to stream the full backend output, e.g. when debugging.

Value

A tibble of segment-level embeddings (one row per segment), or, for model="whisper" with embeddings="both", a named list of two tibbles (encoder, decoder). For several audio files, a named list with one such result per file. The settings used are saved as a comment (retrieve with comment()).

For model = "whisper", feature columns are named f<dimension>_<statistic>: each hidden dimension is summarised across the segment's time frames with five statistics – _mea (mean), _med (median), _var (variance), _min and _max. For example, the default whisper-medium model has 1024 hidden dimensions, so embedding six diarised segments returns a 6 x 5124 tibble: segment_id, start_sec, end_sec and speaker, plus 1024 x 5 = 5120 feature columns (f00000_mea ... f01023_max). The _mea columns alone are the closest analogue to standard mean-pooled embeddings. model = "whispa" instead returns a single WhiSPA embedding per segment (no summary statistics).

Details

Two backends are supported: Whisper (encoder and/or decoder hidden-state summary statistics) and WhiSPA (a single speech-psychological embedding per segment).

See also

talkEmbed, talkTranscribeDiarise.

Examples

if (FALSE) { # \dontrun{
wav_path <- system.file("extdata", "test_short.wav", package = "talk")
diar <- talkTranscribeDiarise(audio = wav_path)
emb <- talkEmbedSegments(
  audio = wav_path,
  transcript = diar,
  model = "whisper",
  embeddings = "encoder"
)
emb
} # }

GitHub