The need for explainability of Deep Neural Network (DNN) models and the development of AI systems that can fundamentally reason has exponentially increased in recent years, especially with the increasing use of AI/ML models in risk-sensitive and safety-critical applications. Causal reasoning helps identify input variables that cause a certain prediction, rather than merely be correlated, and thus provide useful explanations in practice. Similarly, focusing on causal input-output relationships can help a DNN model generalize to out-of-distribution samples better, where spurious correlations in training data may otherwise mislead a model. This talk will introduce the growing field of explainable AI, summarize existing efforts and focus on one important aspect of causality in DNN models -- the notion of causal attributions between input and output variables of the model. We will do this from two perspectives -- firstly, we will study how one can "deduce" what causal input-output attributions an already-trained DNN model has learned, and provide an efficient mechanism to compute such causal attributions (based on our work published at ICML 2019). Secondly, we will explore the complementary side of this problem on how one can "induce" known prior causal information into DNN models during the training process itself (based on our work published at ICML 2022) . Both of these efforts are derived by a first-principles approach to integrating causal principles into DNN models, and can have significant implications on practice in real-world applications.