mirror of
https://github.com/storytold/artcraft.git
synced 2026-11-02 08:28:18 +00:00
Brandon Thomas
2dd5894283
Current runtime err:
```
2019-09-23 01:00:01.189765: W
tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at
transpose_op.cc:157 : Invalid argument: transpose expects a vector of
size 1. But input(1) is a vector of size 3
thread 'main' panicked at 'Run success: {inner:0x5589ea8b3860,
InvalidArgument: transpose expects a vector of size 1. But input(1) is a
vector of size 3
```
Getting there...!
Voder
Voice things.
Current State
- Chunked/batched synthesis from the CycleGAN network is proven to work!
- Desktop PC PulseAudio is borked. Need to get the
feedback.rsdemo working. - Next steps:
- Fix desktop audio so that mic and speaker work
- Integrate Neural Network (or lightweight transform) into
feedback.rs/cpalcode.
Resources
Crates to try:
-
- Encoder/decoder
- Very mature (3.4.0)
-
- Hardware audio out, low level
- Tons of downloads!
-
- Microphone in, speaker out
- ALSA
-
- convert between floats, u8s, u16s, etc. via a trait
- clip amplitude
- scale playback
- iterators
- signal generators (sine, saw, noise, etc.)
-
- generate spectrograms!
- THEY LOOK SICK!
-
- supports range of audio encoders/decoders
- flac, ogg, wav, alac
-
- generate images from wave files
- con "it is probably threadsafe"
-
- sample rate conversion
-
- read and write wav files
- con: old (2016)
-
- con: old (2016)
- con: low on doc
-
- determine pitch and volume of samples
- uses "bistream autocorrelation function (bcf)"
- todo: also investigate BACF, uses peaks instead of zero-crossings
-
- wav library, based on nom zero-copy parser
- PCM support
- 18/16/32/64 bit support
Hardware
Enumerate hardware:
pactl list short sinks
The linux headphone jack is named at startup:
speaker-test -t wav -c 6 -D front
To change the 'default' speaker,
pactl list short sinks
pactl set-default-sink 'alsa_output.pci-0000_00_1f.3.analog-stereo'
speaker-test -t wav -c 6 -D default
This makes the rust code work on Desktop.
Links
Troubleshooting
THIS WORKS TO DEFEAT CHIPMUNK NOISES!
device.serial = "Focusrite_Scarlett_2i4_USB"
device.string = "surround40:1"
device.buffering.buffer_size = "1048576"
device.buffering.fragment_size = "524288"
PulseAudio's default sampling rate and bit depth are set to 44100Hz @ 16 bits.
1048576 / 1411200 (buffer size) 0.7430385487528345 = 743 ms 524288 / 1411200 (fragment size) 0.37151927437641724 = 372 ms
TensorFlow Model
Saved model (.pb file) must be in saved_model/saved_model.pb or similar.
Need TensorFlow model tools:
bazel build tensorflow/python/tools:saved_model_cli
# NB: Save the graph
definition = self.model.sess.graph_def
directory = 'output_model_pb'
tf.train.write_graph(definition, directory, 'model.pb', as_text=False)
# https://github.com/tensorflow/models/issues/3530#issuecomment-395968881
output_dir = './saved_model/'
builder = tf.saved_model.builder.SavedModelBuilder(output_dir)
builder.add_meta_graph_and_variables(self.model.sess, [
tf.saved_model.tag_constants.SERVING],)
builder.save()