Algorithms on billion-scale graph using 10GB RAM: I love DataFusion
semyonsinchenko.github.ioI implemented a graph map-reduce using Apache DataFusion. Where possible, I offloaded everything to disk, and designed the algorithms to rely on bulk scans rather than random access. DataFusion handles spillover, sort-merge joins, aggregations, planning and execution, so my code … Read more