I’m experimenting with a novel approach that integrates symbolic logic directly into a transformer’s attention mechanism. By using a custom spaCy-based logic parser, I generate a “logic mask” that guides the self-attention layers to focus on logical constructs. In preliminary tests with a fine-tuned LLaMA 3 8B model, this method has shown promising improvements on symbolic reasoning tasks (e.g., achieving around 62% on the FOLIO dataset). I’m eager to hear thoughts and suggestions from the community on further refining this approach. Also please note I don’t have a PhD nor masters in machine learning. Happy to take any criticism good or bad. :)<p>https://marqcodes.com/logicEnhanced.html