|
|
Post by tIB on Oct 14, 2023 8:29:07 GMT
tIB Hi Andy, I'm also very sorry to hear you are going through these issues with the TBD... Firstly thanks for diving into this one and for all the background info - Robert L mentioned about an older bug that sounded similar but had been squashed and was wondering if there was a regression of some sort - not being a code guy I can only imagine the complexities of finding something like that. I won't pretend to know enough about flash ROM, though given saving/recall is the issue and it's losing saved presets (computer app side as well as in the unit) it seems likely I'd have thought. Another thing Ive seen btw is that when editting parameters in the edit window, some settings are lost when scrolling beyond where that parameter can be viewed in the edit window. That's been the case on two android devices and my windows machine. (Edit: not 100% on that happening in windows thinking about it - might be that has been android app bug...) I'll try to get some time this weekend to have a fiddle - in terms of building a custom firmware I can have a go with that assuming I can figure it out. I've never found GitHub the most user friendly experience for the non-computer types like me. Backing up the TBD - I didn't know about any of that! I've been using the individual export/import functionality recently, though wasn't even doing that at one point. No issues at all sending that over - is pm the best way or do you want to pm me an email address? Obviously if I back up the recent config the saved settings/ presets are already lost - do you want me to send that over and then another config with the presets/apps reloaded that broke last time? Probably easiest just to let me know exactly what you want there but I've no issues getting whichever is best to you. Firmware wise I can check that too - I guess that's first thing to rule out, though I'm 99% it's the latest one - I've been using Robert L's newer front end recently and that points to the latest revision I think. Pretty sure I cross referenced numbers over at GitH too. I'll pass that on later today... In terms of the shell window I'm usually running unattached to the computer - I set my presets up, save and work untethered - though I can reinstall the last configuration and see if I can break it again quickly. Last time all I did was load plugins/slots until the settings were gone. I'd basically load, listen, load the next save, listen etc... Eventually one would load without CV to pitch and then all the others using the same plugin would also be broken. After one went the other (different) plugins would also get lost soon after. OS wise I'm running Windows here - I think it's 10 on the laptop I use for the TBD. So yeah, thanks for shouting up and thanks to the other Robert in the post above too - I'd love to get these things working in the way I want since when they do they're perfect for my needs. (The all backup might also help me quite a lot too, since I've spent quite a lot of time starting from scratch in the past - one setup and a save/load seems like a lot less work testing wise!) |
|
|
|
Post by tIB on Oct 14, 2023 8:33:13 GMT
Here's my firmware - found in an earlier exchange with Robert. Hopefully you can make that out...  |
|
|
|
Post by tIB on Oct 14, 2023 10:34:57 GMT
I'v registered for GitHub and have had a go at creating a custom firmware... unsuccessfully.
Whenever I look at that site I feel like I'm in some kind of fantasy role playing affair - no clue what I'm doing.
Anyway, afaik I've forked the repository and created a token (whatever that means?!) but for whatever reason the firmware won't compile. (Edit: from the custom firmware link posted above by Robert earlier)
I'm afraid my coding/computing skills are going to be massively out of line with what's needed here, though obviously I'll try as best I can since I'd really like to help with getting issue ironed out.
|
|
|
|
Post by tIB on Oct 14, 2023 11:11:38 GMT
Also can I just ask - in terms of recreating the issue these three modules have, have you guys tested that on the AEM version of the TBD? I can't fathom that I'm that unlucky to have the only three units in AEM that do this?
Last time I tried I was able to lose presets with just repeated loading between three plugins irrc. Took me less than ten minutes to lose settings.
|
|
rma
New Member
Posts: 16
|
Post by rma on Oct 14, 2023 14:17:38 GMT
I have just tried 0.9.21-4-ga9c7f as firmware (comes with the tapp application from github.com/aemodular/ctag-tbd/releasesYes, please make a single document with all steps, which lead to the issue. It may be useful, if you first flash the firmware and bring the TBD into a defined initial state so that we can follow the steps to reproduce. Your document could then look like this: - Flash fresh firmware - Boot TBD - Load Subotnik on Ch0 - Edit patch - Save patch - ... - ... - ... - Crash If it is easier, just summarize your steps here in the forum, I will try to replicate then. Cheers  |
|
|
|
Post by tIB on Oct 14, 2023 14:37:52 GMT
I have just tried 0.9.21-4-ga9c7f as firmware (comes with the tapp application from github.com/aemodular/ctag-tbd/releases)Yes, please make a single document with all steps, which lead to the issue. It may be useful, if you first flash the firmware and bring the TBD into a defined initial state so that we can follow the steps to reproduce. Your document could then look like this: - Flash fresh firmware - Boot TBD - Load Subotnik on Ch0 - Edit patch - Save patch - ... - ... - ... - Crash If it is easier, just summarize your steps here in the forum, I will try to replicate then. Cheers So last time after reflash, from memory I did this: Boot TDB Load subotnik on 0 Edit parameters - CV0 to pitch. Save preset. Snap to module slot (inc. naming) Repeat the above for braids on slot 1 and rains on slot 2 Remove usb Load presets on the device using long press. After about ten minutes of cycling through loads I'd lost the CV to pitch setting on subot. I lost them on the other two shortly after. In this case the module doesn't crash - it simply loses the saved settings. With the crashing that seems to happen after a plugin takes longer than expected to load. I'm not sure if the two things are linked but the losing of saved settings is the bug that is most common/frustrating - when settings are lost the preset is no longer seen in the interface. Hope that makes sense. |
|
|
|
Post by tIB on Oct 14, 2023 14:41:37 GMT
I have just tried 0.9.21-4-ga9c7f as firmware (comes with the tapp application from github.com/aemodular/ctag-tbd/releases)Yes, please make a single document with all steps, which lead to the issue. It may be useful, if you first flash the firmware and bring the TBD into a defined initial state so that we can follow the steps to reproduce. Your document could then look like this: - Flash fresh firmware - Boot TBD - Load Subotnik on Ch0 - Edit patch - Save patch - ... - ... - ... - Crash If it is easier, just summarize your steps here in the forum, I will try to replicate then. Cheers Note that's a different firmware to the one I'm using - see picture. Could it be that Robert L's newer AEM tbd application is throwing out a different firmware with those old flash ROM bugs still present? |
|
rma
New Member
Posts: 16
|
Post by rma on Oct 14, 2023 16:27:10 GMT
Ok, I will have a look at it doing same steps and let you know how it behaves on my end... Thx |
|
|
|
Post by tIB on Oct 14, 2023 16:48:34 GMT
Ok, I will have a look at it doing same steps and let you know how it behaves on my end... Thx Thanks. I'm confused about firmwares - mine shows 0.9-2-gbbdd8 whether I flash with the tapp from GitHub or the AE Modular version from Robert. |
|
rma
New Member
Posts: 16
|
Post by rma on Oct 14, 2023 17:34:33 GMT
Do you use a Mac? I use the Tapp release version, possibly this firmware differs from the ones packaged in the Linux and Windows releases...
|
|
|
|
Post by visuellemusik on Oct 14, 2023 17:46:51 GMT
Hi Andrew, I'll try to address all your questions in your previous posts by answering with a ">>" put in front:
Firstly thanks for diving into this one and for all the background info - Robert L mentioned about an older bug that sounded similar but had been squashed and was wondering if there was a regression of some sort - not being a code guy I can only imagine the complexities of finding something like that. >> Well that bug was due to not enough available space for the preset-parameters on the filesystem on the Flash-Rom. But that got fixed, so if it shows up again there must be a reason like different firmware or some kind of additional factor again causing the problem. But of course similar problems also may happen due to different reasons.
I won't pretend to know enough about flash ROM, though given saving/recall is the issue and it's losing saved presets (computer app side as well as in the unit) it seems likely I'd have thought.
>> Sounds very reasonable to me, too. BTW: the storage is on the unit only, the TBD is running a webserver and your PC/Mac/Linux-box mirror that in your browser.
Another thing Ive seen btw is that when editting parameters in the edit window, some settings are lost when scrolling beyond where that parameter can be viewed in the edit window. That's been the case on two android devices and my windows machine. (Edit: not 100% on that happening in windows thinking about it - might be that has been android app bug...) >> Mh, I never experienced that, this could be either due to the lack of storage (again) or a problem with the webserver on the unit, which also could be due to lack of storage, of course. But it's hard to make a proper guess here unless Robert or I can do proper debugging, or have more data, for instance from that Shell-Window running in the background.
I'll try to get some time this weekend to have a fiddle - in terms of building a custom firmware I can have a go with that assuming I can figure it out. I've never found GitHub the most user friendly experience for the non-computer types like me.
>> I agree to that. But afaik there also is a Web-interface for that app. But tbh I have not tried it out myself, so I am afraid I can't be of much help with that right now.
Backing up the TBD - I didn't know about any of that! I've been using the individual export/import functionality recently, though wasn't even doing that at one point. No issues at all sending that over - is pm the best way or do you want to pm me an email address?
>> I already sent my details via PM.
Obviously if I back up the recent config the saved settings/ presets are already lost - do you want me to send that over and then another config with the presets/apps reloaded that broke last time? Probably easiest just to let me know exactly what you want there but I've no issues getting whichever is best to you.
>> Best would be (as similarly described by Robert M. in this thread already) to a) Have your _correctly working_ environment (the "all.tbd" file) b) The exact steps you did until it crashedc) Possibly a copy/paste of the output in the shell-window (which might be minimized but should still be available)
Firmware wise I can check that too - I guess that's first thing to rule out, though I'm 99% it's the latest one - I've been using Robert L's newer front end recently and that points to the latest revision I think. Pretty sure I cross referenced numbers over at GitH too. I'll pass that on later today...
>> This has been addressed by Robert M. already, so I guess it's better to have this sorted out between the two of you, because he is the expert on that for sure.
In terms of the shell window I'm usually running unattached to the computer - I set my presets up, save and work untethered - though I can reinstall the last configuration and see if I can break it again quickly. Last time all I did was load plugins/slots until the settings were gone. I'd basically load, listen, load the next save, listen etc... Eventually one would load without CV to pitch and then all the others using the same plugin would also be broken. After one went the other (different) plugins would also get lost soon after.
>> Ah ok, get that. So this only is an option in case you experience a crash while working tethered.
OS wise I'm running Windows here - I think it's 10 on the laptop I use for the TBD.
>> Windows 10 still should be completely ok.
So yeah, thanks for shouting up and thanks to the other Robert in the post above too - I'd love to get these things working in the way I want since when they do they're perfect for my needs.
>> Totally get that!
(The all backup might also help me quite a lot too, since I've spent quite a lot of time starting from scratch in the past - one setup and a save/load seems like a lot less work testing wise!) >> That's what I thought, too!
Here's my firmware - found in an earlier exchange with Robert. Hopefully you can make that out... >> I read that like "0.9.21-2-gbbdd8" - otherwise see above please.
I'v registered for GitHub and have had a go at creating a custom firmware... unsuccessfully.
Whenever I look at that site I feel like I'm in some kind of fantasy role playing affair - no clue what I'm doing.
Anyway, afaik I've forked the repository and created a token (whatever that means?!) but for whatever reason the firmware won't compile. (Edit: from the custom firmware link posted above by Robert earlier)
>> As mentioned above I never tried that environment myself, so you would have to ask Robert M. for details about that. But again: there should be (or at least it had been [planned]) a simple to handle webpage for that as well. Also, afaik this process is not part of the core-project for TBD / done by another person with a forked version.
I'm afraid my coding/computing skills are going to be massively out of line with what's needed here, though obviously I'll try as best I can since I'd really like to help with getting issue ironed out.
>> No worries, your input is already very helpful.
Also can I just ask - in terms of recreating the issue these three modules have, have you guys tested that on the AEM version of the TBD? I can't fathom that I'm that unlucky to have the only three units in AEM that do this?
Last time I tried I was able to lose presets with just repeated loading between three plugins irrc. Took me less than ten minutes to lose settings.
>> As mentioned earlier: I had similar situation in the past, but never with the version we finally released for the AEM TBD.The only main difference to your setup is that I always have the TBD connected via USB, but so far I can't think of any reason for a problems when working untethered?
|
|
|
|
Post by tIB on Oct 14, 2023 17:48:55 GMT
Do you use a Mac? I use the Tapp release version, possibly this firmware differs from the ones packaged in the Linux and Windows releases... I'm windows - I can see on the GitHub that the Linux and Mac are later builds - perhaps there's a firmware build in the later packages that differs from the one in the windows release? |
|
|
|
Post by tIB on Oct 14, 2023 17:54:21 GMT
Best would be (as similarly described by Robert M. in this thread already) to a) Have your _correctly working_ environment (the "all.tbd" file) b) The exact steps you did until it crashedc) Possibly a copy/paste of the output in the shell-window (which might be minimized Many thanks for the detailed reply! I'll reflash abd set up tomorrow in the same way I did last time and will set about breaking it with the computer attached... Appreciate the help folks! (I'll reply to your pm shortly too) |
|
rma
New Member
Posts: 16
|
Post by rma on Oct 14, 2023 18:49:25 GMT
What I have done: - downloaded release version of aem tbd tapp for windows from github.com/aemodular/ctag-tbd/releases- with tapp erased flash of tbd, flashed firmware, flashed sample rom, started browser ui - loaded subotnik on ch0 - mapped parameter pitch to cv0 - saved preset - snapped favorite - repeated those steps for MacOsc (Braids) and TBDings (Rings) - exported 3 favorites (attached .jsn files) (in favorites section, click export) - removed USB, changed favorites multiple times with button presses on module - turned power off and on - changed favorites multiple times with button presses on module - reconnected USB - reflashed firmware with tapp and started browser ui - imported favorites from .jsn files, in favorites section, click import - removed USB - changed favorites multiple times with button presses on module - turned power off and on - changed favorites multiple times with button presses on module What I have noticed: - loading the favorite with subotnik takes a little longer, but after like 5-10s it is there - otherwise I have not had any weird behaviour or crashes Your issue seems hard to reproduce, unfortunately  The firmware I have tested now was the one included in tapp windows, i.e. 0.9.21-2-gbbdd0 (same as you have). This is effectively 0.9.21, the remainder of the version string comes from the Github cloud build and should be irrelevant. I had suspected that something with the flash storage gets corrupted when you interact with the module, but given the steps you described and I have tested, the module only stores which plugin was last selected for the channels and which preset was activated. Sorry that I couldn't help with this, for me everything seems to work fine and as expected. Try out my steps above and import the favorites I have attached by clicking import in favorites section. Let me know what happens... Cheers Attachments:braids.jsn (862 B)
subbo.jsn (4.15 KB)
rings.jsn (935 B)
|
|
rma
New Member
Posts: 16
|
Post by rma on Oct 14, 2023 18:50:36 GMT
Ah, btw, I really like your music on bandcamp, great stuff! |
|
|
|
Post by tIB on Oct 14, 2023 19:27:55 GMT
What I have done: - downloaded release version of aem tbd tapp for windows from github.com/aemodular/ctag-tbd/releases- with tapp erased flash of tbd, flashed firmware, flashed sample rom, started browser ui - loaded subotnik on ch0 - mapped parameter pitch to cv0 - saved preset - snapped favorite - repeated those steps for MacOsc (Braids) and TBDings (Rings) - exported 3 favorites (attached .jsn files) (in favorites section, click export) - removed USB, changed favorites multiple times with button presses on module - turned power off and on - changed favorites multiple times with button presses on module - reconnected USB - reflashed firmware with tapp and started browser ui - imported favorites from .jsn files, in favorites section, click import - removed USB - changed favorites multiple times with button presses on module - turned power off and on - changed favorites multiple times with button presses on module What I have noticed: - loading the favorite with subotnik takes a little longer, but after like 5-10s it is there - otherwise I have not had any weird behaviour or crashes Your issue seems hard to reproduce, unfortunately The firmware I have tested now was the one included in tapp windows, i.e. 0.9.21-2-gbbdd0 (same as you have). This is effectively 0.9.21, the remainder of the version string comes from the Github cloud build and should be irrelevant. I had suspected that something with the flash storage gets corrupted when you interact with the module, but given the steps you described and I have tested, the module only stores which plugin was last selected for the channels and which preset was activated. Sorry that I couldn't help with this, for me everything seems to work fine and as expected. Try out my steps above and import the favorites I have attached by clicking import in favorites section. Let me know what happens... Cheers Thanks for trying - I do have backups of the JSN files I was using during the recent testing - I can send those your way tomorrow if that's useful? I can also try these... What I'll do is reflash, reload a full set of the patches I used in the last round of breaking it, do an all backup to pass on to Matthias and then set about messing with those above. Hopefully I can get to that tomorrow... |
|
|
|
Post by tIB on Oct 14, 2023 19:28:21 GMT
Ah, btw, I really like your music on bandcamp, great stuff! Why thank you very much! |
|
rma
New Member
Posts: 16
|
Post by rma on Oct 15, 2023 5:52:01 GMT
Yes, please send me some preset files.
I had the module running over night and started playing again this morning, creating a few more patches.
It seems something similar to your experience happened, it lost a preset from the Braids patch.
I'll keep checking and send an update what I find.
Cheers
|
|
rma
New Member
Posts: 16
|
Post by rma on Oct 15, 2023 6:19:14 GMT
I think it is related to the technical implementation of the filesystem on the ESP32 (the little processor on the TBD), manufactured by Espressif. They have deprecated the SPIFFS file system, which the current TBD firmware uses in order to make file access. Reason being too unreliable Reference here: github.com/RalphBacon/203-SPIFFS-vs-LITTLEFSI will have a look into a remedy, but can't promise when I will be able to get at it, sorry. Possibly switching to LittleFS may help github.com/joltwallet/esp_littlefsWhy I think this is the issue is that when TBD "looses a preset" it defaults back to the initially flashed preset. We implemented this this way, in order to avoid issues in case that a preset write isn't completed properly, i.e. due to a power loss during a flash write cycle. Thanks for being patience! |
|
|
|
Post by tIB on Oct 15, 2023 9:55:03 GMT
Yes, please send me some preset files. I had the module running over night and started playing again this morning, creating a few more patches. It seems something similar to your experience happened, it lost a preset from the Braids patch. I'll keep checking and send an update what I find. Cheers That's the one - what I found was that when one preset was lost, all presets with the same plugin were also affected. Seeing as you seen to have recreated the issue do you still need files? |
|
|
|
Post by tIB on Oct 15, 2023 10:01:05 GMT
I think it is related to the technical implementation of the filesystem on the ESP32 (the little processor on the TBD), manufactured by Espressif. They have deprecated the SPIFFS file system, which the current TBD firmware uses in order to make file access. Reason being too unreliable Reference here: github.com/RalphBacon/203-SPIFFS-vs-LITTLEFSI will have a look into a remedy, but can't promise when I will be able to get at it, sorry. Possibly switching to LittleFS may help github.com/joltwallet/esp_littlefsWhy I think this is the issue is that when TBD "looses a preset" it defaults back to the initially flashed preset. We implemented this this way, in order to avoid issues in case that a preset write isn't completed properly, i.e. due to a power loss during a flash write cycle. Thanks for being patience! To the untrained eye that sounds like quite a big problem/rewrite? I'm glad you've managed to recreate it though - fingers crossed it isn't asich work as it sounds. Good luck! |
|
|
|
Post by visuellemusik on Oct 15, 2023 10:37:21 GMT
Yes, please send me some preset files. I had the module running over night and started playing again this morning, creating a few more patches. It seems something similar to your experience happened, it lost a preset from the Braids patch. I'll keep checking and send an update what I find. Cheers That's the one - what I found was that when one preset was lost, all presets with the same plugin were also affected. Seeing as you seen to have recreated the issue do you still need files? As you say "when one preset was lost, all presets with the same plugin were also affected", I am not entirely sure if that fits to the fallback-behaviour of the SPIFFS file system, because then again a stable, previous state should be reestablished. Also it took Robert an overnight wait until the error happened. So yes, I guess it still would be helpful to have a deeper look at your environment / "all.jsn" file and the steps you take until it crashes. At least it should speed up the process to recreate the issue. Of course this won't help with switching to a completely different underlying filesystem, but maybe we could find some kind of bypass-solution, at least for an intermediate fix. But very much like Robert M. I can't promise anything, it's all about options ;-) |
|
|
|
Post by tIB on Oct 15, 2023 11:18:16 GMT
As you say "when one preset was lost, all presets with the same plugin were also affected", I am not entirely sure if that fits to the fallback-behaviour of the SPIFFS file system, because then again a stable, previous state should be reestablished. I suppose a more accurate way to say that is 'when I reloaded other patches using the same plugin they had also lost settings' - I think that's because when it loses a preset/parameters it also loses any saved presets on the plugin, so the other presets/variations/saves are not found when it tries to load that save. Sorry if that's unlcear - I find the terminology on this module with plugins, presets and user favourites (?!) tricky! Ask if unclear. In fairness, when the replacement module arrived and first lost presets I set about just trying to break it each time, just to check and double check the bug. Last time out i did it in well under ten minutes, though I didn't count how many times I asked it to load in that time. Not a normal use case though, and in the past it's lasted for varying lengths of time, though never really long enough to be useful. I actually put the rig away for an extended period of time while waiting for the replacement module (when robert and I felt it was hardware related). It's probably also worth mentioning about the crash/reboot loop - while it's probably a separate issue to the lost presets, could that also relate to the same file system issue? Regardless, given that is much less common that the losing of presets I guess that's one to be lived with with a reflash, asssuming the all.jsn import works fine - I guess I'll just back up and reflash/load if that crops up again. Losing presets/saves is a bit more difficult to live with though. Hopefully there's something that can solve that - I'm open to trying a custom firmware with just the plugins I use in if that still seemslike an idea that might work better(?), though I wasn't able to create that in github yesterday. |
|
|
|
Post by visuellemusik on Oct 15, 2023 17:35:38 GMT
Well concerning the loop-situation: as mentioned before, I once experienced that too, but really a long time ago before we officially released the TBD package for AE Modular. What I could see in the serial monitor back then was that it was looking for a file that should have been there but could not be found. So at least in my case this also seemed very much filesystem related.
I then assumed that the backup filesystem already had been corrupted before, because: Afaik it works like this: at some point in time the stored files are assumed to be alright and then are copied to a backup-location. I'm assuming (without actually knowing, sorry) this happens upon boot. Then during a new session (everything that happens after the boot) all presets are stored to the primary (current) filesystem and if something goes terribly wrong, the system tries to restore the current filesystem from the backup one. If that also fails, it reaches an unstable state and boots. If on bootup the primary filesystem is corrupt, it again tries to restore it from the backup, if that fails this goes on and on, hence the reboot-loop.
But so far (without any information from the serial monitor and/or debugging the situation properly) at least I personally really can't tell what's going on, so there is a lot of speculation and more or less educated guessing going on. So let's see what we can find out (if anything) looking at your specific configuration, which I'll start right now ;-) |
|
|
|
Post by tIB on Oct 15, 2023 18:24:40 GMT
Well concerning the loop-situation: as mentioned before, I once experienced that too, but really a long time ago before we officially released the TBD package for AE Modular. What I could see in the serial monitor back then was that it was looking for a file that should have been there but could not be found. So at least in my case this also seemed very much filesystem related.
I then assumed that the backup filesystem already had been corrupted before, because: Afaik it works like this: at some point in time the stored files are assumed to be alright and then are copied to a backup-location. I'm assuming (without actually knowing, sorry) this happens upon boot. Then during a new session (everything that happens after the boot) all presets are stored to the primary (current) filesystem and if something goes terribly wrong, the system tries to restore the current filesystem from the backup one. If that also fails, it reaches an unstable state and boots. If on bootup the primary filesystem is corrupt, it again tries to restore it from the backup, if that fails this goes on and on, hence the reboot-loop.
But so far (without any information from the serial monitor and/or debugging the situation properly) at least I personally really can't tell what's going on, so there is a lot of speculation and more or less educated guessing going on. So let's see what we can find out (if anything) looking at your specific configuration, which I'll start right now ;-) Great stuff. With the boot loop that happens infrequently enough for it to be 'one of those things' as far as I'm concerned - now I know how to make/upload an all.jsn I'll reflash and do that if/when it breaks in that way. (As an aside I think I'd tried to do the backup/reinstall before but had assumed it hadn't worked due to the way the tapp doesn't show the changes until reboot.) The preset losing is a bigger issue for me, in that it happened much more frequently. As for the debugging/serial monitor stuff sorry I can't really help more there - I can try with it tethered to usb/shell window if you can't recreate the issue with the presets I sent... Let me know if you need me to. |
|
