Any advice or tips for dealing with incrementing values?

[digby8]

I’m working on a new plugin (currently just designing how data will be stored) & i think it’s probably going to work using a database of rows that equal actions & every 3 or 5 minutes it’ll go through a few of the rows & do their action, this is because the actions are mostly just incrementing values & i don’t want multiple actions being preformed on the same database row as i’m guessing it could result in only one update going through by overwriting the other update.

Why do i think i need to worry about this: my plugin will be triggering & updating the same row(s) within seconds of each other very often & in unpredictable & changing frequencies.

I do already have a lot of ideas on how to make this database of actions that gets run consistently:
Flow with trigger ‘every minute’
because it can run a minute early or late, track in a database row the last time run & if it’s been more then 60 seconds then run again, if not don’t run & wait to next trigger.
Will use ‘get rows’ block i guess to go through them, check time every iteration & compare to time when flow started, if time is close (maybe 2 to 3 seconds) to 15 seconds since flow started (no way to know if bot has premium or not as far as i know but maybe i can make it a plugin setting) to exit flow as to not exit flow mid-iteration.

One big problem is with users spamming actions like triggers ‘reaction add’ & then ‘reaction remove’ which i don’t want to clog up the upcoming action list database. i could just make it search for rows for ‘reaction add’ & delete them when getting the trigger ‘reaction remove’ but that would still very rarely delete a row when it’s currently being acted on & then the update that it should be reverted never gets acted on but this would be very very rare so it’s probably no issue, the bigger issue with this idea is that it’s pretty complex but probably still what i’ll go with.

This topic/post isn’t looking for the solution & won’t be closed for such. This is just for discussion, ideas & any advice on this so if you have any thoughts then they’d likely be useful for me & maybe even others to hear.
Hope this all makes sense, sorry for any mistakes i might have made & i can try to clarify anything if anyone really needs me to.

[magicbotman]

@digby8

i don’t want multiple actions being preformed on the same database row as i’m guessing it could result in only one update going through by overwriting the other update.

This is correct, in general Inventor handles DB race conditions by only persisting the last value.

@digby8

my plugin will be triggering & updating the same row(s) within seconds of each other very often & in unpredictable & changing frequencies.

This is usually fine as long as you can accept the system being somewhat lossy. Keep in mind that Discord is a flawed interface for any system with highly concurrent access anyway (given how interactions are sent and received, rapid message edits are disallowed, etc.) so this isn’t the sort of thing we optimize for in designing the platform.

@digby8

I do already have a lot of ideas on how to make this database of actions that gets run consistently:
Flow with trigger ‘every minute’
because it can run a minute early or late, track in a database row the last time run & if it’s been more then 60 seconds then run again, if not don’t run & wait to next trigger.

You can do this, but in general we’re discouraging the use of the every minute trigger in plugins unless you have a good reason. You haven’t explained what the plugin does, that context would be needed to determine if it’s the sort of thing we would allow.

I wouldn’t worry too much about the minute early/late thing. Average variance is on the order of seconds. We will occasionally have periods of high server load and switch to running those jobs every other minute. Most importantly, the runtime tries very hard to ensure those jobs have at least 45 seconds (max execution time) worth of padding between them, to prevent exactly this kind of problem.

Important: If you are reading this in the far future, do note that this is subject to change, especially as we work to increase maximum flow run times and improve how recurring jobs are handled. The banner on the block is always going to give you the safest advice to ensure your flow doesn’t break with future updates, which may change these undocumentd behaviors.

@digby8

Will use ‘get rows’ block i guess to go through them, check time every iteration & compare to time when flow started, if time is close (maybe 2 to 3 seconds) to 15 seconds since flow started (no way to know if bot has premium or not as far as i know but maybe i can make it a plugin setting) to exit flow as to not exit flow mid-iteration.

This is correct, a setting might be the way to go. We may be allowing plugins to have a fixed (larger than free) exec time in the future regardless of the bot it is installed in, as a flag that can be requested/approved during review if needed. Perf impact still needs to be evaluated, so no promises.

@digby8

One big problem is with users spamming actions like triggers ‘reaction add’ & then ‘reaction remove’ which i don’t want to clog up the upcoming action list database. i could just make it search for rows for ‘reaction add’ & delete them when getting the trigger ‘reaction remove’ but that would still very rarely delete a row when it’s currently being acted on & then the update that it should be reverted never gets acted on but this would be very very rare so it’s probably no issue, the bigger issue with this idea is that it’s pretty complex but probably still what i’ll go with.

You could have a dedicated table for per-user action ratelimits, basically just storing a user ID and timestamp of last action, with a custom block that drops/exits flows if the action was less than a few secs ago, or updates the timestamp if it wasn’t. There’s potential for race conditions but those would only allow temporarily bypassing the ratelimit and not result in any other data being corrupted.

You should also consider using buttons if at all possible, since Discord handles the rate-limiting on those in a way that is a lot less problematic if you’re looking to do a lot of stuff in response to the press.

I hope at least some of that helps.

[zammer8]

@digby8

Flow with trigger ‘every minute’
because it can run a minute early or late, track in a database row the last time run & if it’s been more then 60 seconds then run again, if not don’t run & wait to next trigger.

That will make it more wrong, since it could be up to a few minutes off now.

What plugin is this for?

[digby8]

@zammer8

That will make it more wrong, since it could be up to a few minutes off now.

I don’t need it to be ran every minute, just at least one minute apart.

@zammer8

What plugin is this for?

Don’t wanna say yet, sorry for the inconveniences because of that.

[zammer8]

@digby8

Don’t wanna say yet, sorry for the inconveniences because of that.

That does make things a lot more inconvenient, since this could be an XY Problem

[digby8]

@magicbotman thanks for all the info, even just the confirmation for things is really helpful for me.

@magicbotman

You can do this, but in general we’re discouraging the use of the every minute trigger in plugins unless you have a good reason. You haven’t explained what the plugin does, that context would be needed to determine if it’s the sort of thing we would allow.

Well i need a flow to be triggered often to do through the database rows, do their action & then delete the row. I need it to be triggered every certain amount of time so that i can know for sure it’s only running one at a time. There’ll also be likely a lot of rows & the flow won’t be able to go through all of them in 15 or even 30/45 seconds, so it’s probably best if this time trigger is on the shorter end & i don’t know if 15 minutes is short enough, something like 5 minutes might work but there’s no trigger for that so i think every minute is best.

@magicbotman

I wouldn’t worry too much about the minute early/late thing. Average variance is on the order of seconds. We will occasionally have periods of high server load and switch to running those jobs every other minute. Most importantly, the runtime tries very hard to ensure those jobs have at least 45 seconds (max execution time) worth of padding between them, to prevent exactly this kind of problem.

That’s really great to hear! I won’t worry or plan for that happening then.

@magicbotman

This is correct, a setting might be the way to go. We may be allowing plugins to have a fixed (larger than free) exec time in the future regardless of the bot it is installed in, as a flag that can be requested/approved during review if needed. Perf impact still needs to be evaluated, so no promises.

Interesting, plugins are already very overpowered but that would be very helpful for consistency in knowing how your plugin will be used, i’m sure you’ve already thought about all of this though so i won’t go on.

@magicbotman

You could have a dedicated table for per-user action ratelimits, basically just storing a user ID and timestamp of last action, with a custom block that drops/exits flows if the action was less than a few secs ago, or updates the timestamp if it wasn’t. There’s potential for race conditions but those would only allow temporarily bypassing the ratelimit and not result in any other data being corrupted.

You should also consider using buttons if at all possible, since Discord handles the rate-limiting on those in a way that is a lot less problematic if you’re looking to do a lot of stuff in response to the press.

Maybe i’m misunderstanding but my problem isn’t with a user doing something like sending lots of messages (each that add a row to the DB) but specifically with discord reactions (which i must use to make what i want to, buttons or anything else won’t work at all) & with someone adding then removing a reaction again & again which with how i’m planning to make this plugin, would create a lot of DB rows that are all just to revert what other rows would do but i can’t just not accept one of these discord actions as then i have a row that says someone added a reaction when they have since removed it.

@magicbotman

I hope at least some of that helps.

Definitely! Sorry i don’t want to say much about what my plugin is/does but i just don’t want the stress of others knowing what i’m making, which wasn’t great for me or the end product when it came to my giveaway plugin. I hope you can understand.

[magicbotman]

@digby8

something like 5 minutes might work but there’s no trigger for that so i think every minute is best.

We’ve floated 5 minute trigger before, but it seems more likely that every minute will just be turned into every 2-5 minutes once percice scheduled jobs become available (for things like giveaways) and don’t need to be handled by frequently recurring jobs anymore.

@digby8

Maybe i’m misunderstanding but my problem isn’t with a user doing something like sending lots of messages (each that add a row to the DB) but specifically with discord reactions (which i must use to make what i want to, buttons or anything else won’t work at all) & with someone adding then removing a reaction again & again which with how i’m planning to make this plugin, would create a lot of DB rows that are all just to revert what other rows would do but i can’t just not accept one of these discord actions as then i have a row that says someone added a reaction when they have since removed it.

You will need to accept it being lossy as a given, and work from there to find the solution that is most reliable. Keep in mind that Discord gateway events aren’t guaranteed to arrive at all or in any particular order.

If your plugin would consider spamming reactions to be an abuse case, losing that data shouldn’t be problematic. If you are designing your plugin for users to be expected to spam reactions, that also likely isn’t something we would approve given the library rules on abuse of Discord’s API - we would consider encouraging reaction spam to be encouraging abuse.

I tend to agree with Zammer that your vagueness about what exactly this plugin does precludes us helping in a useful capacity and this really does smell like an XY problem situation. I’m not sure there’s much else we’ll be able to help with here if you aren’t willing to divulge more details about what you are actually building.