This tool is pretty amazing - if youre looking for a crawler with 1 million tokens for freem, works direct from console/python/whatever - and I built an HN comment summarizor and crawler html —> .md output in 30 seconds.
(I can put in in a gist if you like)
(Then I started adding function to extract all my browsing history, categorize it, allow me to seach it, recrawl the page/search the page and if find it, save out those sections… but the context gets to long in a single file - so I am currently breaking all that out…
(Just a fun thing to do… (because you can use Jina.ai’s embeddor as i want to make model trained on all my comment history across my browsing history, and all my accounts.)
I want an toolset that is specific to all the sites, accounts and categories I follow…)
To use these tools
To make my own little “web experience timeline”
This is the number one person I like to follow in this space: