Allow extension to run for more time? | XM Community
Solved

Allow extension to run for more time?

  • 16 April 2024
  • 2 replies
  • 19 views
JesperAndersen
Rank
Userlevel 3
Badge +11

Hi @qHubert,

I’m not sure where to direct this question, so I’m starting with you :)

Is there any way it would be possible to have an extension run for more than the default 45 seconds?

In short, I’m building an extension that would need to process a lot of users in a project, and perform various tasks on them. In my current code, I had to build in limits to stop the code from processing too many users at once, as this would mean the code would hit the 45 second limit and then fail.

This means that I can fairly safely process 250 users in 45 seconds, but if the server is slow, then I can risk that this would not even work. Running this every hour would give me the ability to process 6000 users per day, but in some cases this will not be enough.

Worst case scenario, the solution in that case would be to split this into multiple projects, based on country or similar, and then just set up 8 projects with 8 connected workflows that would then load 8 jobs at the same time on the servers. This would increase the capacity to a theoretical 48.000 users, but loads in the projects would be very uneven, so might just be closer to 20-30.000 users per day.

I get that one job should not block the server for an extended amount of time but seeing that the SF extractor can run for 10 mins, makes me think that other jobs should be able to run a bit longer than the 45 seconds?.

Connecting the tasks might fix this a bit, but still not really. If the first tasks attempt to get all users in a project, and it is not able to do so within 45 seconds, should the next task then continue collecting users, or should it try to process those from the previous task, which might not be possible, if it failed due to running for too long? You see my point?

Btw, either I missed it completely or the documentation on connecting tasks is missing? There is a short bit in the plugin-app.jsx file, but only on the export, not how you pick up files from other tasks?

Thanks in advance!

 

 

 

icon

Best answer by qHubert 19 April 2024, 21:51

View original

2 replies

qHubert
Rank
Userlevel 1
Badge +5

Hey there,

I’m assuming that you’re looking into XM Functions to do this, hence the 45s time limit. Unfortunately, this is a hard limit that we cannot increase so we’ll have to find some other way of doing what you want to do. 

Generally speaking there are two main paradigms when it comes to developing for workflows. We have the long-running workflows like the SFSF extractor that operate on larger datasets, and we have more event-driven workflows using a webservice task that operates on a single record. While XM Functions allows you to work somewhere in between, it isn’t designed for the long running large dataset use cases. 

On the Export function, it’s simply to export a calculated value to the next task. In the next task, if it has an option to pipe data, you’ll be able to choose the export from a previous task as well. That’s how you ingest to another task. 

What I’d recommend is to come to an office hours with my team. From your profile, I see that you’re part of QPN. Can you please use the QPN assistant and request office hours with our developer relations team? This probably warrants a more holistic discussion on what you can do. 

JesperAndersen
Rank
Userlevel 3
Badge +11

Thanks for the feedback (again) @qHubert. This is truly appreciated!

I have added a request for office hours and sent a long description of the challenge.

After reviewing your suggestion on the linking of tasks, I have split the extension up in multiple tasks, but passing large arrays of users between two tasks will eat up some of the precious 45 seconds I have in each task, and that just requires even more tasks :) 

I would suggest to add the passing of data in the documentation or have I missed that?

Leave a Reply


Terms & Conditions