mirror of
https://github.com/storytold/artcraft.git
synced 2026-11-04 01:18:34 +00:00
200 lines
6.2 KiB
Markdown
200 lines
6.2 KiB
Markdown
Voder
|
|
=====
|
|
|
|
Voice things.
|
|
|
|
Current State
|
|
-------------
|
|
|
|
* Chunked/batched synthesis from the CycleGAN network is proven to work!
|
|
* Desktop PC PulseAudio is borked. Need to get the `feedback.rs` demo working.
|
|
* Next steps:
|
|
* Fix desktop audio so that mic and speaker work
|
|
* Integrate Neural Network (or lightweight transform) into `feedback.rs/cpal` code.
|
|
|
|
|
|
TensorFlow
|
|
----------
|
|
|
|
### Tensorflow bindings (do they know things?)
|
|
I had to do a thing in the Cargo.toml,
|
|
```
|
|
[patch.crates-io]
|
|
# NB: Manually upgraded 'tensorflow-sys' library from 1.13.1 to 1.14.0
|
|
# The downloaded tensorflow libs in target/ must be moved to /lib
|
|
# (I couldn't figure out the Rust linker flags)
|
|
tensorflow = { path = "/home/bt/dev/3rd/tensorflow.rs" }
|
|
```
|
|
|
|
The `libtensorflow.so.1` might go missing if dependencies change. I've been copying it
|
|
around like a lazy person that doesn't understand the build process. Gross.
|
|
|
|
```
|
|
+ bt@halide:~/dev/voder/target/debug/build/tensorflow-sys-0fdab7e44f6d3e04/out$ ls -lA
|
|
-r-xr-xr-x 1 bt bt 34748520 Sep 30 01:55 libtensorflow_framework.so
|
|
-r-xr-xr-x 1 bt bt 216546752 Sep 30 01:55 libtensorflow.so
|
|
|
|
+ bt@halide:~/dev/voder/target/debug/build/tensorflow-sys-0fdab7e44f6d3e04/out$ cp ~/dev/voder/target/debug/build/tensorflow-sys-b3beaea4e172b1c8/out/libtensorflow.so.1 .
|
|
+ bt@halide:~/dev/voder/target/debug/build/tensorflow-sys-0fdab7e44f6d3e04/out$ cp ~/dev/voder/target/debug/build/tensorflow-sys-b3beaea4e172b1c8/out/libtensorflow.so.1.14.0 .
|
|
+ bt@halide:~/dev/voder/target/debug/build/tensorflow-sys-0fdab7e44f6d3e04/out$ cp ~/dev/voder/target/debug/build/tensorflow-sys-b3beaea4e172b1c8/out/libtensorflow_framework.so.1 .
|
|
+ bt@halide:~/dev/voder/target/debug/build/tensorflow-sys-0fdab7e44f6d3e04/out$ cp ~/dev/voder/target/debug/build/tensorflow-sys-b3beaea4e172b1c8/out/libtensorflow_framework.so.1.14.0 .
|
|
+ bt@halide:~/dev/voder/target/debug/build/tensorflow-sys-0fdab7e44f6d3e04/out$ ls -lA
|
|
|
|
-r-xr-xr-x 1 bt bt 34748520 Sep 30 01:55 libtensorflow_framework.so
|
|
-r-xr-xr-x 1 bt bt 34748520 Oct 11 00:31 libtensorflow_framework.so.1
|
|
-r-xr-xr-x 1 bt bt 34748520 Oct 11 00:31 libtensorflow_framework.so.1.14.0
|
|
-r-xr-xr-x 1 bt bt 216546752 Sep 30 01:55 libtensorflow.so
|
|
-r-xr-xr-x 1 bt bt 216546752 Oct 11 00:30 libtensorflow.so.1
|
|
-r-xr-xr-x 1 bt bt 216546752 Oct 11 00:31 libtensorflow.so.1.14.0
|
|
|
|
(And now it works. Ugh)
|
|
```
|
|
|
|
### Python Model Output
|
|
|
|
- sf1tm1.ckpt.meta
|
|
- ~10MB
|
|
- Graph structure definition. Not needed if you still have the Python code.
|
|
```python
|
|
with tf.Session() as sess:
|
|
saver = tf.train.import_meta_graph('/tmp/model.ckpt.meta')
|
|
saver.restore(sess, "/tmp/model.ckpt")
|
|
```
|
|
- https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/protobuf/meta_graph.proto
|
|
- https://stackoverflow.com/a/36203288
|
|
- index file
|
|
- a string-string immutable table(tensorflow::table::Table). Each key is a name of a tensor and its
|
|
value is a serialized BundleEntryProto. Each BundleEntryProto describes the metadata of a tensor:
|
|
which of the "data" files contains the content of a tensor, the offset into that file, checksum,
|
|
some auxiliary data, etc.
|
|
https://stackoverflow.com/a/45033500
|
|
|
|
|
|
### Model
|
|
Saved model (.pb file) must be in `saved_model/saved_model.pb` or similar.
|
|
|
|
Need TensorFlow model tools:
|
|
|
|
`bazel build tensorflow/python/tools:saved_model_cli`
|
|
|
|
```
|
|
# NB: Save the graph
|
|
definition = self.model.sess.graph_def
|
|
directory = 'output_model_pb'
|
|
tf.train.write_graph(definition, directory, 'model.pb', as_text=False)
|
|
|
|
# https://github.com/tensorflow/models/issues/3530#issuecomment-395968881
|
|
output_dir = './saved_model/'
|
|
builder = tf.saved_model.builder.SavedModelBuilder(output_dir)
|
|
builder.add_meta_graph_and_variables(self.model.sess, [
|
|
tf.saved_model.tag_constants.SERVING],)
|
|
builder.save()
|
|
```
|
|
|
|
Resources
|
|
---------
|
|
|
|
Crates to try:
|
|
|
|
- [hound](https://crates.io/crates/hound)
|
|
- Encoder/decoder
|
|
- Very mature (3.4.0)
|
|
|
|
- [cpal](https://crates.io/crates/cpal)
|
|
- Hardware audio out, low level
|
|
- Tons of downloads!
|
|
|
|
- [wavy](https://crates.io/crates/wavy)
|
|
- Microphone in, speaker out
|
|
- ALSA
|
|
|
|
- [sample](https://crates.io/crates/sample)
|
|
- convert between floats, u8s, u16s, etc. via a trait
|
|
- clip amplitude
|
|
- scale playback
|
|
- iterators
|
|
- signal generators (sine, saw, noise, etc.)
|
|
|
|
- [sonogram](https://crates.io/crates/sonogram)
|
|
- generate spectrograms!
|
|
- THEY LOOK SICK!
|
|
|
|
- [audrey](https://crates.io/crates/audrey)
|
|
- supports range of audio encoders/decoders
|
|
- flac, ogg, wav, alac
|
|
|
|
- [waveform](https://crates.io/crates/waveform)
|
|
- generate images from wave files
|
|
- con "it is probably threadsafe"
|
|
|
|
- [samplerate](https://crates.io/crates/samplerate)
|
|
- sample rate conversion
|
|
|
|
- [riff-wave](https://crates.io/crates/riff-wave)
|
|
- read and write wav files
|
|
- con: old (2016)
|
|
|
|
- [wavefile](https://crates.io/crates/wavefile)
|
|
- con: old (2016)
|
|
- con: low on doc
|
|
|
|
- [pitch](https://crates.io/crates/pitch)
|
|
- determine pitch and volume of samples
|
|
- uses "bistream autocorrelation function (bcf)"
|
|
- todo: also investigate BACF, uses peaks instead of zero-crossings
|
|
|
|
- [libwave](https://crates.io/crates/libwave)
|
|
- wav library, based on nom zero-copy parser
|
|
- PCM support
|
|
- 18/16/32/64 bit support
|
|
|
|
|
|
Hardware
|
|
--------
|
|
|
|
Enumerate hardware:
|
|
|
|
* `pactl list short sinks`
|
|
|
|
The linux headphone jack is named at startup:
|
|
|
|
* `speaker-test -t wav -c 6 -D front`
|
|
|
|
To change the 'default' speaker,
|
|
|
|
```
|
|
pactl list short sinks
|
|
pactl set-default-sink 'alsa_output.pci-0000_00_1f.3.analog-stereo'
|
|
speaker-test -t wav -c 6 -D default
|
|
```
|
|
|
|
This makes the rust code work on Desktop.
|
|
|
|
Links
|
|
-----
|
|
|
|
* http://www.rossbencina.com/code/real-time-audio-programming-101-time-waits-for-nothing
|
|
|
|
Troubleshooting
|
|
---------------
|
|
|
|
THIS WORKS TO DEFEAT CHIPMUNK NOISES!
|
|
|
|
* https://bbs.archlinux.org/viewtopic.php?id=179923
|
|
* https://wiki.archlinux.org/index.php/PulseAudio/Troubleshooting#Setting_the_default_fragment_number_and_buffer_size_in_PulseAudio
|
|
|
|
```
|
|
device.serial = "Focusrite_Scarlett_2i4_USB"
|
|
device.string = "surround40:1"
|
|
device.buffering.buffer_size = "1048576"
|
|
device.buffering.fragment_size = "524288"
|
|
```
|
|
PulseAudio's default sampling rate and bit depth are set to 44100Hz @ 16 bits.
|
|
|
|
> 1048576 / 1411200 (buffer size)
|
|
0.7430385487528345 = 743 ms
|
|
> 524288 / 1411200 (fragment size)
|
|
0.37151927437641724 = 372 ms
|
|
|
|
|