Voder

Voice things.

Current State

Chunked/batched synthesis from the CycleGAN network is proven to work!
Desktop PC PulseAudio is borked. Need to get the feedback.rs demo working.
Next steps:
- Fix desktop audio so that mic and speaker work
- Integrate Neural Network (or lightweight transform) into feedback.rs/cpal code.

TensorFlow

Tensorflow bindings (do they know things?)

I had to do a thing in the Cargo.toml,

[patch.crates-io]
# NB: Manually upgraded 'tensorflow-sys' library from 1.13.1 to 1.14.0
# The downloaded tensorflow libs in target/ must be moved to /lib 
# (I couldn't figure out the Rust linker flags)
tensorflow = { path = "/home/bt/dev/3rd/tensorflow.rs" }

The libtensorflow.so.1 might go missing if dependencies change. I've been copying it around like a lazy person that doesn't understand the build process. Gross.

+ bt@halide:~/dev/voder/target/debug/build/tensorflow-sys-0fdab7e44f6d3e04/out$ ls -lA
-r-xr-xr-x 1 bt bt  34748520 Sep 30 01:55 libtensorflow_framework.so
-r-xr-xr-x 1 bt bt 216546752 Sep 30 01:55 libtensorflow.so

+ bt@halide:~/dev/voder/target/debug/build/tensorflow-sys-0fdab7e44f6d3e04/out$ cp ~/dev/voder/target/debug/build/tensorflow-sys-b3beaea4e172b1c8/out/libtensorflow.so.1 . 
+ bt@halide:~/dev/voder/target/debug/build/tensorflow-sys-0fdab7e44f6d3e04/out$ cp ~/dev/voder/target/debug/build/tensorflow-sys-b3beaea4e172b1c8/out/libtensorflow.so.1.14.0 . 
+ bt@halide:~/dev/voder/target/debug/build/tensorflow-sys-0fdab7e44f6d3e04/out$ cp ~/dev/voder/target/debug/build/tensorflow-sys-b3beaea4e172b1c8/out/libtensorflow_framework.so.1 . 
+ bt@halide:~/dev/voder/target/debug/build/tensorflow-sys-0fdab7e44f6d3e04/out$ cp ~/dev/voder/target/debug/build/tensorflow-sys-b3beaea4e172b1c8/out/libtensorflow_framework.so.1.14.0 . 
+ bt@halide:~/dev/voder/target/debug/build/tensorflow-sys-0fdab7e44f6d3e04/out$ ls -lA

-r-xr-xr-x 1 bt bt  34748520 Sep 30 01:55 libtensorflow_framework.so
-r-xr-xr-x 1 bt bt  34748520 Oct 11 00:31 libtensorflow_framework.so.1
-r-xr-xr-x 1 bt bt  34748520 Oct 11 00:31 libtensorflow_framework.so.1.14.0
-r-xr-xr-x 1 bt bt 216546752 Sep 30 01:55 libtensorflow.so
-r-xr-xr-x 1 bt bt 216546752 Oct 11 00:30 libtensorflow.so.1
-r-xr-xr-x 1 bt bt 216546752 Oct 11 00:31 libtensorflow.so.1.14.0

(And now it works. Ugh)

Model

Saved model (.pb file) must be in saved_model/saved_model.pb or similar.

Need TensorFlow model tools:

bazel build tensorflow/python/tools:saved_model_cli

# NB: Save the graph
definition = self.model.sess.graph_def
directory = 'output_model_pb'
tf.train.write_graph(definition, directory, 'model.pb', as_text=False)

# https://github.com/tensorflow/models/issues/3530#issuecomment-395968881
output_dir = './saved_model/'
builder = tf.saved_model.builder.SavedModelBuilder(output_dir)
builder.add_meta_graph_and_variables(self.model.sess, [
	tf.saved_model.tag_constants.SERVING],)
builder.save()

Resources

Crates to try:

hound
- Encoder/decoder
- Very mature (3.4.0)
cpal
- Hardware audio out, low level
- Tons of downloads!
wavy
- Microphone in, speaker out
- ALSA
sample
- convert between floats, u8s, u16s, etc. via a trait
- clip amplitude
- scale playback
- iterators
- signal generators (sine, saw, noise, etc.)
sonogram
- generate spectrograms!
- THEY LOOK SICK!
audrey
- supports range of audio encoders/decoders
- flac, ogg, wav, alac
waveform
- generate images from wave files
- con "it is probably threadsafe"
samplerate
- sample rate conversion
riff-wave
- read and write wav files
- con: old (2016)
wavefile
- con: old (2016)
- con: low on doc
pitch
- determine pitch and volume of samples
- uses "bistream autocorrelation function (bcf)"
- todo: also investigate BACF, uses peaks instead of zero-crossings
libwave
- wav library, based on nom zero-copy parser
- PCM support
- 18/16/32/64 bit support

Hardware

Enumerate hardware:

pactl list short sinks

The linux headphone jack is named at startup:

speaker-test -t wav -c 6 -D front

To change the 'default' speaker,

pactl list short sinks
pactl set-default-sink 'alsa_output.pci-0000_00_1f.3.analog-stereo'
speaker-test -t wav -c 6 -D default

This makes the rust code work on Desktop.

Troubleshooting

THIS WORKS TO DEFEAT CHIPMUNK NOISES!

https://bbs.archlinux.org/viewtopic.php?id=179923
- https://wiki.archlinux.org/index.php/PulseAudio/Troubleshooting#Setting_the_default_fragment_number_and_buffer_size_in_PulseAudio

device.serial = "Focusrite_Scarlett_2i4_USB"
device.string = "surround40:1"
device.buffering.buffer_size = "1048576"
device.buffering.fragment_size = "524288"

PulseAudio's default sampling rate and bit depth are set to 44100Hz @ 16 bits.

1048576 / 1411200 (buffer size) 0.7430385487528345 = 743 ms 524288 / 1411200 (fragment size) 0.37151927437641724 = 372 ms

5.4 KiB Raw Blame History