Open Forum

Expand all | Collapse all

RapidStart Export Limits

  • 1.  RapidStart Export Limits

    Posted 22 days ago
    My company has been using RapidStart for roughly a year and half. There have been a couple of instances where we have run into a message stating that 'the file is too large for export' which usually happens because we are trying to pull too many columns resulting in too much data.

    We had an odd situation this past week and were hoping for some clarification/extra insight. I am not approaching this from an engineering perspective so the more layman terms the better:

    Our goal was to pull the Sales Shipments Header table for various pieces of information. In total we needed to export 12 columns including the one required Validate Field.

    To test that the information was correct, we first exported a 12 column Configuration Package from our TEST database. Our TEST database, like most others, is quite slow so the export took roughly a 20ish minutes but we were able to successfully export all 311,000 database records in the applicable 12 columns.

    Once we confirmed the information was correct we attempted to export the same Configuration Package from our PROD database which had 315,000 database records. This is where we first received 'the file is too large' error message. As we have done before, we split the configuration package into 2 separate Configuration Packages (one with 6 columns, one with 7 columns) and attempted to export them at separate times. Again, we received the same error message on both Configuration Packages. Eventually, I was able to successfully export the same table with only 3 columns (i used a different method to gather the needed information).

    The question we have is: Is it common for a PROD database to have different limitations on exported data than a TEST database? While there was certainly more data to export from PROD it doesn't appear to be a massive amount more. The puzzling part is when we shaved nearly half the columns off the export we still received the error.

    Any insight that can be offered is much appreciated. If there is something we can do to help with the export performance in PROD we want our IT staff to take a look at that.

    ------------------------------
    Nick Frisk
    Analyst & Service Project Coordinator
    TricorBraun Flex, formerly Pacific Bag
    Woodinville WA
    ------------------------------


  • 2.  RE: RapidStart Export Limits

    TOP CONTRIBUTOR
    Posted 22 days ago
    There are some values on the NAV Service Tier you could have your IT team look at.

    On the Client Services Tab there is one called Max Upload Size (I know you are exporting, but it affects both). You could try increasing that. The service tier needs to be restarted afterwards, so it is best to do that after hours.

    As for the speed, it's just going to take a while to export 300,000 records. If you're only occasionally doing things of this size, and aren't experiencing other performance issues day to day, I wouldn't worry about it too much.  Even if you could cut it in half, the amount of man hours it would take and / or potential hardware costs likely outweigh saving 10 minutes a few times a year.

    ------------------------------
    Matt Traxinger
    NAV Developer
    ArcherPoint Inc.
    San Antonio TX
    ------------------------------



  • 3.  RE: RapidStart Export Limits

    Posted 22 days ago
    Thanks Matt!

    Our IT Director took a look at this and did note that there was a significant difference in the MAX Upload Size (900MB in TEST vs 100MB in PROD).

    Even though the final excel file from TEST is only 25MB he assumes the size of data in NAV is larger. Is that a correct assumption?

    ------------------------------
    Nick Frisk
    Analyst & Service Project Coordinator
    TricorBraun Flex, formerly Pacific Bag
    Woodinville WA
    ------------------------------



  • 4.  RE: RapidStart Export Limits

    TOP CONTRIBUTOR
    Posted 22 days ago
    Honestly I'm not sure, but I have read of other cases where it seemed the value on the server configuration was above the range and it still gave the same error. Increasing it further allowed the export to go through.

    ------------------------------
    Matt Traxinger
    NAV Developer
    ArcherPoint Inc.
    San Antonio TX
    ------------------------------



  • 5.  RE: RapidStart Export Limits

    Posted 22 days ago
    Understood. Thanks again for the help here.

    ------------------------------
    Nick Frisk
    Analyst & Service Project Coordinator
    TricorBraun Flex, formerly Pacific Bag
    Woodinville WA
    ------------------------------



  • 6.  RE: RapidStart Export Limits

    Posted 21 days ago
    Hi Nick,

    I have worked with Rapid Start, and I found it interesting the you split the package by columns. My instinct would have been to split the package by rows instead by using a date filter. You would just want to be careful not to accidentally include a duplicate date or exclude a date all together.

    I am not sure, but that might have a bigger impact on reducing the file size by reducing the total number of records your exporting.

    I hope that helps.

    ------------------------------
    Paul Davis
    The Sun Valley Rice Co., LLC
    Dunnigan CA
    ------------------------------



  • 7.  RE: RapidStart Export Limits

    Posted 20 days ago
    Thanks for this Paul!

    It is interesting that you bring up a date filter on the RapidStart export. It's something my manager and I have been looking for and it was our first instinct as we are used to filtering other tables in NAV. However, we have not been able to figure out how to use date filters on the configuration package. Can you elaborate on that more?

    ------------------------------
    Nick Frisk
    Analyst & Service Project Coordinator
    TricorBraun Flex, formerly Pacific Bag
    Woodinville WA
    ------------------------------



  • 8.  RE: RapidStart Export Limits

    Posted 20 days ago
    Hi Nick,

    Happy to help.

    First, there are two different filters on configuration packages, which can be confusing.

    The most obvious filter is in the ribbon of the sub-table, between the "Find" and "Clear Filter" actions. This is not the filter that you want to use.

    Instead, with the table containing the field that you want to filter on selected, click the "Table" action that opens the drop-down menu. Below "fields" is "filters". This is what you want.

    When you click here it will open the Config. Package Filters page. This page will be blank, with no records. You want to enter a record by choosing the field from the table you have selected that you would like to filter on. In your case, I would typically filter Sales Shipment Header by Posting Date.

    Lastly, on the same line as the field you selected, is a column called "Field Filter." It is here where you want to enter your date filter. For example I just exported every record since the start of the year with >=1/1/2019, and I also tested 1/1/2019.. both syntaxes returned the same results.

    I hope that helps!  #DataManagement

    Best,​​​

    ------------------------------
    Paul Davis
    The Sun Valley Rice Co., LLC
    Dunnigan CA
    ------------------------------



  • 9.  RE: RapidStart Export Limits

    Posted 20 days ago
    This is awesome.

    I see where you as saying you have to be careful with the date ranges/filters used. Thanks again for this insight, it will certainly help with the size of files, especially on projects (such as the one we completed this week) that only need a certain date range to be included.

    ------------------------------
    Nick Frisk
    Analyst & Service Project Coordinator
    TricorBraun Flex, formerly Pacific Bag
    Woodinville WA
    ------------------------------



  • 10.  RE: RapidStart Export Limits

    TOP CONTRIBUTOR
    Posted 19 days ago
    Nick - The out of the box setting for exporting data to Excel is set to 5,000 lines limit. Just to let everyone know you can change this by using the Dev environment and changing the value. It is under the Tools>Options tab and the value is called Max. no. of XML records to send - default value you will see is 5,000


    ------------------------------
    John Grant
    Senior Application Consultant
    Innovia Consulting
    ------------------------------



If you've found this thread useful, dive deeper into User Group community content by role