Fix remove ignored files from import

[JeremyFabrikapp]

Hey !
During import from local folder, the script load all files including node_modules, env (stored in cookies) and .gitignored files.
As import MAX_FILES is 1000 it may quickly get stuck.

I've made a quick fix to avoid this, that would allow to upload folder more efficiently, by removing npm/git/ignored files from import.

[thecodacus]

this is nice but why not use the ignore library to ignore the paths that match the pattern, I am not confident on the glob to regex conversion. I recommend using https://www.npmjs.com/package/ignore

[aliasfoxkde]

My suggestion would be to maintain your project on GitHub and just pull in the repo with the latest changes. That way best practice is to use .gitignore (by default) and that would resolve this issue. I think the idea behind importing a folder is for prototyping something really quick, not uploading Gigs of data to the Web Container. And while it's not a horrible idea (maybe make bolt use the same .gitignore file when importing), I'm not sure we can support this. I think it's more or a procedure and maybe provide a warning, than really a thing to fix in code. Just my two cents. Thanks

[wonderwhy-er]

Ignore is already used here

bolt.diy/app/utils/fileUtils.ts

Line 42 in 1f938fc

return !ig.ignores(path);

Its imported into
https://github.com/stackblitz-labs/bolt.diy/blob/main/app/components/chat/ImportFolderButton.tsx#L38

I remember testing that it was filtered out.
Need to retest.
But your regex code is not needed, as @aliasfoxkde said ignore library and existing code should be used.

[JeremyFabrikapp]

Hi, thanks for your input! Yes basically the gitignore is taken in account after the files are have been counted, so I guess it's just a matter of moving that check point so that's the gitignore is taken in account while counting files right after file upload. The issue currently is that uploading a folder with a few files but a node_modules folder will trigger that max files error, while those ignored files are then removed and not sent to workbench. I'll give it another look for a smoother integration, thanks for your remarks ! Le lun. 23 déc. 2024, 11:07, Eduard Ruzga ***@***.***> a écrit :

…

Ignore is already used here https://github.com/stackblitz-labs/bolt.diy/blob/1f938fca407aa7fe024fb3d835abb97a285a3621/app/utils/fileUtils.ts#L42 Its imported into https://github.com/stackblitz-labs/bolt.diy/blob/main/app/components/chat/ImportFolderButton.tsx#L38 I remember testing that it was filtered out. Need to retest. But your regex code is not needed, as @aliasfoxkde <https://github.com/aliasfoxkde> said ignore library and existing code should be used. — Reply to this email directly, view it on GitHub <#783 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASV2XSC7V45FZWGDHQEXYH32G7OFDAVCNFSM6AAAAABTXESZUGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJZGM2TIMRRGY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[clintebbesen]

I also came across this issue. Tried to import a project that has about 5 files containing code, but got blocked as the total files was over 3000 in the node_modules directory

[leex279]

@clintebbesen can you provide this project? I just tried with my task-list-advanced project which as over 12000 files with node_modules and it importet without problems (newest version of bolt)

[clintebbesen]

I tried to import it again now and it's giving the same message for me, but a lot worse now haha, over 40,000 files. 40,909 files in node_modules.

[leex279]

@clintebbesen just to be sure. you are on the latest release 0.0.6 Version? And you are on a Mac?