Spaces:
Running on Zero
Great job! :)
@yalishanda Hey Alex!
Thank you for creating and sharing this model! It's very interesting and I was wondering if I can ask you a few questions?
- Why did you go with MDN? Did it show a better performance?
- Have you considered using masked encoder-based archtecture? From my experience, it performs really well usually...
- Have you considered training on more data? E (GMD) are nice and everything but IMHO they are too small to train a well performing model.
So, if you get a minute, I wanted to invite you to check out my Discover MIDI Dataset. It has ~1M+ solo drum track MIDIs which (with included filtering) can be used for large drum track model pretraining.
Also, check out my Bidi Score where I use masked encoders to inpaint individual MIDI notes elements (i.e velocities or timings)
Last but not least I wanted to suggest giving your models more context to work with. I.e drum track + lead monophonic melody so that the model can infer velocities easier.
Anyways, thank you again for your models and the demo. Its really nice otherwise :)
I would be interested to chat about all of this if you have free time :)
Let me know.
Sincerely,
Alex.
Hi!
Thank you for your interest and recommendations!
- MDN is one of a few heads that I experimented with, in fact the categorical head transformer showed a bit better dynamic range preservation and MAE score than MDN so it may sound often better. - I added it to the model selector to try out Also I noticed that MDN still misses a lot of accents, especially on bigger temperature settings (try with something closer to 0)
- The transformer backbone I use for both MDN and categorical is already using a masked encoder architecture. The specific bit here is that velocity here is only an output and never an input to other notes, so I try to predict all velocities in one pass (100% fixed mask rate). Maybe in the future I can experiment with some partial velocity input from neighboring notes, but the structural-only context here for those is on purpose.
- Yes, future research and development should definitely involve finding and training on more data. This demo here is only after a few hours of training and experimenting :) Thank you for the dataset and models recommendations, great work, I will for sure check them out! If there are non-flat (performed by a human on a kit) midi drums inside the Discover MIDI dataset they will be perfect for the next generation of training :)
As for more context, the melody is a nice idea, although not always applicable. One other thing I have in mind is a selector for general "loudness" that the model 's output should be centered around. The partial velocity input is another thing to try.
The models sound still quite choppy and LightGBM has the best MAE, although a very thin dynamic range, so some balance needs to be found between it and the transformer models.
I see you have a lot of projects related to MIDI which is impressive! This is a topic of interest to me as well so I am very glad we can exchange ideas and possibly even collaborate :) My goal is to eventually submit a paper on this research (after I use to complete my Master's thesis which I am now writing 😀 )
Best regards,
Alex
Thank you for your response and feedback :) Also, thank you for the follow and for liking my MIDI projects. It means a lot to me :)
I've realized that I did not provide you with links to my drum projects which you may find interesting in terms of implementations and output. So here they are:
https://huggingface.co/spaces/asigalov61/Ultimate-Drums-Transformer
https://huggingface.co/spaces/projectlosangeles/Awesome-Drums-Transformer
The first one is rather the legacy project, but it still performs really well and produces nice output in most cases.
And the second one is more recent implementation of the same but more advanced.
However, these are not encoder-based which is why they sometimes do not perform well with some input compositions.
I really want to make an encoder-based one (which would be similar to yours) for solid drum track inpainting which is why I was interested in your project because I wanted to implement some of your ideas.
From my experience, masked encoders-based architectures (especially with iterative refinement) show best performance for such tasks. Primarily because encoders can see the whole sequence/composition unlike AR decoder-only ones, which helps to produce a human-like/believable drum track. However, the most important part here is the context. Symbolic music models perform best when given a nice, structured context to go with so providing it to the model in some form is very important for quality of output IMHO.
And sure, I am always open to collaborating, especially if it involves symbolic music/MIDI projects. So, I would be more than happy to share my knowledge and experience with you which you are welcome to use in your project or for your master thesis :)
Take a look at those two of my drum projects and if you like them and/or have any questions, we can go from there.
Otherwise, thank you for clarifications about MDN and your projects architecture as I was wondering about it.
Sincerely,
Alex.
@yalishanda Hey Alex!
Just pinging you because I wanted to invite you to check out my finished drum track mastering project which was inspired by some of your ideas:
https://huggingface.co/spaces/projectlosangeles/Drum-Track-Master
If you can take a look at it whenever you have a free minute, I would really appreciate it.
And any feedback, thoughts, or criticism is very welcome as well :)
Thanks.
Alex